Skip to content

OIDC token retry under concurrent deploys + cloudExtras.topologySpreadConstraints#329

Merged
Cre-eD merged 2 commits into
mainfrom
feat/oidc-retry-topology-spread
Jun 17, 2026
Merged

OIDC token retry under concurrent deploys + cloudExtras.topologySpreadConstraints#329
Cre-eD merged 2 commits into
mainfrom
feat/oidc-retry-topology-spread

Conversation

@Cre-eD

@Cre-eD Cre-eD commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Two independent reliability/scheduling improvements, kept as two commits for review and rollback:

1. fix(security): retry GitHub Actions OIDC token request under load

Keyless-signing acquired the GitHub Actions OIDC token with a single HTTP GET and a hard 10s client timeout. When several stacks that share one image deploy concurrently from the same push, each process races for a token and the OIDC endpoint intermittently times out (requesting OIDC token: ... context deadline exceeded), failing the deploy.

  • Retry with exponential backoff + full jitter (4 attempts, 1s→8s) to decorrelate concurrent callers.
  • Replace the fixed client timeout with a per-attempt context deadline (20s) that still honours parent-context cancellation (sleeps select on ctx.Done()).
  • Retry only transient failures (transport/timeout, 408, 429, 5xx); fail fast on 4xx and on a malformed/empty 200 body.
  • Overridable via SC_OIDC_TOKEN_REQUEST_ATTEMPTS / SC_OIDC_TOKEN_REQUEST_TIMEOUT.

Acquiring one token per job and sharing it via SIGSTORE_ID_TOKEN (already short-circuited) would also help, but that's a workflow-level change outside this repo; this is the in-repo defense.

2. feat(k8s): cloudExtras.topologySpreadConstraints

Adds a topologySpreadConstraints field to cloudExtras so a stack can spread its replicas across nodes/zones.

On GKE Autopilot this is the cost-correct alternative to pod anti-affinity for node-level HA: Autopilot enforces a 0.5 vCPU minimum request on pods that use pod (anti-)affinity (and rejects pods below it), whereas topology spread is not subject to that minimum — replicas spread at their normal request.

  • Threaded through both cloudExtrasDeploymentConfig converters (kubernetes-native ToKubernetesRunConfig and GKE Autopilot ToGkeAutopilotConfig) and into the pod spec.
  • Conversion split into a pure normalize (defaults + validation) and a mechanical Pulumi mapping:
    • maxSkew default 1 (must be ≥1); whenUnsatisfiable default DoNotSchedule (must be DoNotSchedule|ScheduleAnyway); minDomains only valid with DoNotSchedule; labelSelector defaults to the deployment's own pods.
config:
  scale: { min: 2, max: 4 }
  cloudExtras:
    topologySpreadConstraints:
      - topologyKey: "kubernetes.io/hostname"
        maxSkew: 1
        whenUnsatisfiable: "DoNotSchedule"
        minDomains: 2

On Autopilot, DoNotSchedule + minDomains: 2 is required to actually force a second node; ScheduleAnyway only expresses a preference and lets the scheduler bin-pack both replicas onto one node.

Tests

  • pkg/security: retry success/5xx/429/timeout/exhaustion, fail-fast on 4xx, no-retry on malformed/empty 200, parent-cancel-during-backoff abort, audience-URL building, env short-circuit.
  • pkg/clouds/...: normalize defaults/validation table, no-input-mutation, multiple constraints; threading guards on both converters (the k8s-native and Autopilot paths, which have drifted apart historically); ConvertDescriptor deserialization.

All touched packages: go build, go vet, go test, gofmt clean.

Rollout

After release, consumers bump their pinned SC version to pick up both fixes. The topology-spread change is opt-in (no behavior change unless a stack sets the field).

Cre-eD added 2 commits June 17, 2026 12:11
The keyless-signing OIDC token was fetched with a single HTTP GET and a
hard 10s client timeout. When several stacks that share one image deploy
concurrently from the same push, each process races for a token and the
GitHub Actions OIDC endpoint intermittently times out, surfacing
"requesting OIDC token: context deadline exceeded" and failing the deploy.

Retry the request with exponential backoff and full jitter (4 attempts,
1s->8s) to decorrelate concurrent callers, replace the fixed client
timeout with a per-attempt context deadline (20s), and abort the loop
immediately when the parent context is cancelled (returning ctx.Err()
unwrapped). Only retry transient failures (transport/timeout, 408, 429,
5xx); fail fast on 4xx and on a malformed or empty 200 body. Attempts and
per-attempt timeout are overridable via SC_OIDC_TOKEN_REQUEST_ATTEMPTS /
_TIMEOUT.

Sharing one token across the racing processes would be strictly better but
is a workflow-level change outside this package; this is the in-repo
defense.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Add a topologySpreadConstraints field to cloudExtras so a stack can spread
its replicas across nodes/zones. On GKE Autopilot this is the cost-correct
way to achieve node-level HA: pod anti-affinity forces a 0.5 vCPU minimum
request per pod (Autopilot rejects pods below it), while topology spread is
not subject to that minimum, so replicas spread at their normal request.

The constraint is threaded through both cloudExtras->DeploymentConfig
converters (kubernetes-native and GKE Autopilot) and into the pod spec.
Conversion is split into a pure normalize step (defaults + validation:
maxSkew>=1 default 1, whenUnsatisfiable default DoNotSchedule, minDomains
>=1 and only with DoNotSchedule, labelSelector defaults to the deployment's
own pods) and a mechanical Pulumi mapping, so the defaulting/validation is
unit tested without resolving Pulumi outputs. Threading guards cover both
converters since they copy cloudExtras field-by-field and have drifted
apart before.

To force a second node on Autopilot use whenUnsatisfiable: DoNotSchedule
with minDomains: 2; ScheduleAnyway only expresses a preference and lets the
scheduler bin-pack replicas onto one node.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@github-actions

Copy link
Copy Markdown

Semgrep Scan Results

Repository: api | Commit: 580d083

Check Status Details
⚠️ Semgrep Warning 1 warning(s), 1 total

Scanned at 2026-06-17 08:13 UTC

@github-actions

Copy link
Copy Markdown

Security Scan Results

Repository: api | Commit: 580d083

Check Status Details
✅ Secret Scan Pass No secrets detected
✅ Dependencies (Trivy) Pass 0 total (no critical/high)
✅ Dependencies (Grype) Pass 0 total (no critical/high)
📦 SBOM Generated 522 components (CycloneDX)

Scanned at 2026-06-17 08:14 UTC

@Cre-eD
Cre-eD merged commit 0b990aa into main Jun 17, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants